Skip to content

row/H3-VISION-SCATTER: scatter vision tower into DiT conditioning (deepstack + merged scatter) - #90

Closed
localai-bot wants to merge 1 commit into
mainfrom
row/H3-VISION-SCATTER
Closed

row/H3-VISION-SCATTER: scatter vision tower into DiT conditioning (deepstack + merged scatter)#90
localai-bot wants to merge 1 commit into
mainfrom
row/H3-VISION-SCATTER

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Mission — scatter the loaded vision tower into the DiT conditioning; fix the ref2va grid (spec §8.9)

After #86 the real visual.* tower LOADS + RUNS and image-conditioned fl2va is COHERENT via the VAE-keyframe path, but ref2va gridded because vision features never reached the DiT: the encoder's DEVICE text forward had no deepstack arg and nothing scattered merged vision features into prompt_embeds.

Results

  • deepstack_visual_indexes = {8,16,24} CONFIRMED vs the release config (H3's text_encoder/ IS Qwen3-VL-32B-Instruct: vision_config.deepstack_visual_indexes = [8,16,24], depth 27). H3 image/video-conditioned generation: wire the encoder vision tower to real weights (row/H3-CONDITIONED-E2E) #86 inference was correct.
  • Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice now takes visual_pos_mask + per-tap deepstack and injects into the first N layers (mirrors host reference + upstream _deepstack_process). Merged masked_scatter into inputs_embeds stays the caller's job (upstream _encode). Gate: device==host max|diff| 3.8e-4, DeepStack moves the conditioning. All encoder/vision gates green.
  • Driver --cond-image routes an image through the encoder vision path (ExpandImagePlaceholders + Qwen3VLGetRopeIndex == H3 _get_rope_index for t==1). Additive.
  • Deliverable 3 — fl2va + --cond-image = COHERENT + matching (PASS). GB10: frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). Vision conditioning is SOUND.
  • Deliverable 2 — ref2va + --cond-image STILL GRIDS (honest FAIL, recorded).

Verdict / re-attribution

The "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. The ref2va grid is NOT the encoder conditioning: DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward. fl2va PINS output rows; ref2va PREPENDS free-running reference rows — the one difference.

Records: spec §8.9 + §8.2, STATUS/BENCHMARKS/FEATURES, benchmark-record, NOW, state.

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys

…Stack→device text tower) + ref2va re-attribution

Closes the #86 framework residual and re-attributes the ref2va grid with a GB10
render A/B (row/H3-VISION-SCATTER).

deepstack_visual_indexes CONFIRMED (was #86-inferred): {8,16,24}, grounded in the
release config — MiniMax-H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose
vision_config.deepstack_visual_indexes = [8,16,24], depth 27, text num_hidden_layers
64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig default and public
Qwen/Qwen3-VL-30B-A3B. No value change; comment updated.

Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice
now takes the optional visual_pos_mask + per-tap deepstack blocks and ADDS each block
into the masked visual rows after each of the first len(deepstack) decoder layers —
device mirror of the gated host reference and upstream _deepstack_process
(encoder.py:770-800). The merged masked_scatter into inputs_embeds stays the caller's
job (upstream _encode, encoder.py:1071). Text-only prompts pass the defaults and are
byte-identical. Gate: the device keep-quant encoder test now also runs WITH a visual
mask + two DeepStack blocks → device==host max|diff| 3.8e-4 (<=2e-3) AND DeepStack
moves the conditioning (scale 1.006->1.062).

Driver --cond-image routes a reference image through the encoder vision path (reuse
only: Qwen3VLImageProcessor -> Qwen3VLVisionForward -> merged + 3 DeepStack;
ExpandImagePlaceholders inserts nm image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex
== H3 _get_rope_index for t==1, position math verified). Additive.

GB10 render A/B (256x256/22f/12steps):
* Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS):
  frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved
  toward the prompt). No grid. The vision conditioning is SOUND.
* Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL):
  same multicolour patch grid as #86's text-only ref2va.

RE-ATTRIBUTION (evidence): the "vision-enriched conditioning fixes the grid"
hypothesis is REFUTED. NOT the encoder conditioning — DiT math byte-exact (#74/#77),
the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only
vs vision-enriched prompts. The only difference between coherent fl2va and gridding
ref2va: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running reference rows.
Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages
VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout +
un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward.

Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record,
NOW, state.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
@mudler
mudler force-pushed the row/H3-VISION-SCATTER branch from c7b0271 to e7a9a59 Compare August 7, 2026 02:17
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Landed on main as the squash (mudler authorship). ref2va's reference-row assembly diagnostic runs next.

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys

mudler added a commit that referenced this pull request Aug 7, 2026
…a conditions via the encoder (#90)

row/H3-VISION-SCATTER squash. deepstack_visual_indexes {8,16,24}
CONFIRMED against the release config (the text_encoder IS
Qwen3-VL-32B-Instruct) + vllm-omni defaults. Wired 1:1 with upstream:
per-layer DeepStack scatter-add into masked rows on the DEVICE text
forward (minimax_h3_encoder_device.cpp:103,216-243, mirroring
_deepstack_process encoder.py:770-800); merged masked_scatter into
inputs_embeds at the caller per upstream _encode; driver --cond-image
runs ProcessImage -> vision tower -> placeholder expansion -> scatter
-> rope indexes (byte-equivalent to H3's _get_rope_index at t==1).
Text-only path byte-identical. Gate: device==host 3.8e-4 with DeepStack
proven to move the conditioning (1562 asserts).

GB10 renders: fl2va + --cond-image = COHERENT and MATCHING, evolving
toward the text prompt (keyframe cat -> windowsill in warm sunlight).
ref2va STILL GRIDS — the vision-conditioning hypothesis is REFUTED
(same vision path renders coherent fl2va; grid invariant to text-only
vs vision-enriched). The re-attributed residual: fl2va PINS output rows
while ref2va PREPENDS free-running reference rows — the ref2va
reference-row conditioning ASSEMBLY (VAE-reference rows, noised-anchor
layout, un-pinned target denoise) is the remaining suspect; next
diagnostic = target-row VAE-input adjacency cosine + condition-noise
A/B.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
@localai-bot localai-bot closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants